Skip to content

NV TensorRT RTX EP - initial commit #24456

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 31 commits into from
Apr 24, 2025
Merged

NV TensorRT RTX EP - initial commit #24456

merged 31 commits into from
Apr 24, 2025
+6,615 −26

Conversation

ankan-ban
Copy link
Contributor

@ankan-ban ankan-ban commented Apr 17, 2025 •
edited
Loading

New EP - currently based on existing TensorRT EP but meant to be used on RTX GPUs with a lean version of TensorRT.

Description

Adding a new EP based on TensorRT EP. This is going to use a special version of TensorRT optimized for RTX GPUs. In the future we plan to make changes to the EP to streamline it further (e.g, get rid of dependency on CUDA EP completely).

Motivation and Context

The new TensorRT for RTX is going to have:

  1. Much smaller footprint
  2. Much faster model compile/load times.
  3. Better usability in terms of use of cached models across multiple RTX GPUs.

This effort is also targeting WCR ML workflows.

ankan-ban and others added 2 commits April 17, 2025 17:40
@ankan-ban
NV TensorRT RTX EP - initial commit
fb6a731 
New EP - currently based on existing TensorRT EP but meant to be used on RTX GPUs with a lean version of TensorRT.
@gedoensmax
make CUDA Minimal the default for NV EP
27db0d7
@ankan-ban ankan-ban marked this pull request as draft April 17, 2025 13:54
github-advanced-security[bot]
github-advanced-security bot found potential problems Apr 17, 2025
View reviewed changes
@gaugarg-nv
Fix memory paging issue seen with large models.
b730d49 
Unload the model once it is no longer needed.

Bug: 5225623
@jywu-msft
Copy link
Member

please address lintrunner failure
(clangformat) https://github.com/microsoft/onnxruntime/actions/runs/14519273916/job/40745967030?pr=24456

ankan-ban and others added 19 commits April 18, 2025 07:41
@ankan-ban
Merge pull request #3 from gaugarg-nv/nv-tensorrt-rtx-ep
ac8b694 
Fix memory paging issue seen with large models.
Fix: Apply clang-format formatting
e8b9aca
added NvProviderFactory::CreateProvider(const OrtSessionOptions& sess…
14986ad 
…ion_options, const OrtLogger& session_logger)
added the implementation of SessionOptionsAppendExecutionProvider for
a33d103 
 NV TensorRt Rtx Ep
Fixed the help of onnxruntime_perf_test
90ca9e5
@hrishikeshm
Add support for python bindings of NV TensorRT RTX EP
0267fb2
fixed review comments
718aade
@ankan-ban
Merge pull request #5 from ishwar-raut1/ishwar/nv-tensorrt-rtx-ep
57bee11 
Ishwar/nv tensorrt rtx ep
@ankan-ban
Merge pull request #6 from hrishikeshm/nv-tensorrt-rtx-ep
9a838a5 
Add support for python bindings of NV TensorRT RTX EP
@ankan-ban
use setShapeValuesV2
8c04376 
use setShapeValuesV2 with 64 bit types to support latest TRT RTX builds.
@ankan-ban
Merge pull request #7 from ankan-ban/misc-fixes
e87e5cd 
use setShapeValuesV2
@ankan-ban
Clean up old APIs and options
25312aa 
Also fix a couple of bugs to ensure the options are actually passed down.
@ankan-ban
Merge pull request #9 from ankan-ban/options-clean-up
7dfa562 
Clean up old APIs and options
@ankan-ban
fix formatting
d8a6ce5
@gedoensmax
Significantly reduce options
b998bfb
@ankan-ban
Merge pull request #10 from ankan-ban/minor-formatting-fixes
228f9ce 
fix formatting
@ankan-ban
Merge branch 'microsoft:main' into nv-tensorrt-rtx-ep
3f34076
@gedoensmax
add naiive test
39608f2
@gedoensmax
better warning
62cd19c
@jywu-msft
Copy link
Member

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows GPU Doc Gen CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows x64 QNN CI Pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 5 pipeline(s).

@jywu-msft jywu-msft requested a review from chilo-ms April 23, 2025 04:17
@jywu-msft
Copy link
Member

@chilo-ms please review.

@ankan-ban
Merge pull request #11 from gedoensmax/gedoensmax/nv-reduce-code
38bf50a 
Adds some testing infrastructure and removes lots of deprecated options
@ankan-ban ankan-ban marked this pull request as ready for review April 23, 2025 04:34
@gedoensmax
Copy link
Contributor

@chilo-ms We will happily take any guidance on how to run more test using NV EP to find remaining bugs or implementation gaps.

@chilo-ms
Copy link
Contributor

chilo-ms commented Apr 23, 2025 •
edited
Loading

@chilo-ms We will happily take any guidance on how to run more test using NV EP to find remaining bugs or implementation gaps.

We need to add a new pipeline/CI for building and testing this new NV EP which this PR hasn't done yet.
Considering this new TRT binary is not public yet, some questions here:

  • In terms of building the NV EP, just to double check, i assume those header files from public TRT, i.e. NvInferXXX.h, can be used. (update) setShapeValuesV2 seems to be a new API not yet in NvInferRuntime.h yet.

  • For running NV EP, our Windows pipeline uses A10, can NV EP run on A10? From what i heard it's limited to ampere+ gpus or 40xx/50xx gpus?

  • We might need to upload the private TRT binary to our internal blob storage and let new CI to fetch from there.

@chilo-ms
Copy link
Contributor

To add a new NV EP pipeline in GitHub Action, please duplicate from TRT EP's first and then modify accordingly.

@jywu-msft
Copy link
Member

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows x64 QNN CI Pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 5 pipeline(s).

chilo-ms
chilo-ms approved these changes Apr 24, 2025
View reviewed changes
Copy link
Contributor

@chilo-ms chilo-ms left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code in ORT that makes this new NV EP known to ORT looks good to me.
As for EP code and test/validation is not the focus for this PR, we can discuss later.

@jywu-msft jywu-msft merged commit 2a09f27 into microsoft:main Apr 24, 2025
70 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@snnn snnn snnn left review comments

@ishwar-raut1 ishwar-raut1 ishwar-raut1 left review comments

@adrianlizarraga adrianlizarraga adrianlizarraga left review comments

@RyanUnderhill RyanUnderhill RyanUnderhill left review comments

@gedoensmax gedoensmax gedoensmax left review comments

@jywu-msft jywu-msft jywu-msft approved these changes

@chilo-ms chilo-ms chilo-ms approved these changes

Assignees
No one assigned
Labels
release:1.22.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@ankan-ban @jywu-msft @gedoensmax @chilo-ms @snnn @ishwar-raut1 @adrianlizarraga @RyanUnderhill @gaugarg-nv @hrishikeshm